Towards Real-Time Natural Language Corrections for Assistive Robots

We propose a generalizable natural language interface that allows users to provide corrective instructions to an assistive robotic manipulator in real-time. Allowing human operators to modify properties of how their robotic counterpart achieves a goal on-the-fly increases the utility of the system by incorporating the strengths of the human partner (e.g., visual acuity and environmental knowledge). Our natural language interface is based on a probabilistic graphical model, specifically a Distributed Correspondence Graph (DCG), that is employed to assign semantic meaning to user utterances in the context of the robots environment. We then use the desired corrections to alter the behavior of the robotic manipulator by treating the modifications as constraints on the motion generation (planning) paradigm. In this paper, we highlight four dimensions along which a user may wish to correct the behavior of his or her assistive manipulator. We develop our language model using data collected from Amazon Mechanical Turk in hopes of capturing a comprehensive selection of terminology that real people use to describe desired corrections. To demonstrate the efficacy of our approach, we run a pilot study on hardware with users unfamiliar with robotic systems and analyze points of failure and future directions.